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Summary 


• Evaluation  requirements  for  the  Telemental  Health  Service  (TMH)  operated 
by  the  Alberta  Mental  Health  Board  changed  as  this  program  evolved  from  a 
pilot  project  to  a mature  telehealth  network. 

• A guideline  for  assessment  of  telehealth  applications,  developed  by  the 
Alberta  Heritage  Foundation  for  Medical  Research,  has  been  used  by  the 
AMHB  in  its  evaluation  of  the  TMH. 

• From  a health  technology  assessment  perspective,  most  attributes  referred  to 
in  the  guideline  have  been  well  covered  in  evaluations  of  telemental  health 
undertaken  by  the  AMHB.  However,  there  were  limitations  in  regard  to 
assessment  of  outcomes  and  cost  effectiveness. 

• From  the  perspective  of  the  AMHB,  the  assessment  guideline  has  been 
helpful,  though  more  so  in  the  earlier  stages  of  the  TMH  than  for  appraisal  of 
a mature  network. 

• It  has  not  proved  possible  for  the  AMHB  to  obtain  all  the  measures  of 
performance  suggested  by  the  guideline,  which  does  not  fully  reflect  local 
operational  conditions. 

• A variety  of  assessment  activities  related  to  the  TMH  continue  to  be 
undertaken  by  the  AMHB.  These  have  been  helpful  in  local  decision  making. 

• Constraints  on  assessment  of  the  mature  Telemental  Health  Service  are  the 
complexity  of  the  current  network,  limited  resources  for  evaluation  and 
routine  administrative  requirements  of  decision  - makers. 
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Introduction 


Telehealth  has  had  a long  history  and  interest  in  its  application  continues  to 
grow.  However,  there  is  still  relatively  little  good  quality  information  on  the 
costs  and  benefits  of  telehealth  applications,  particularly  in  the  context  of  routine 
services  k Much  of  the  literature  describes  feasibility  or  pilot  studies,  rather  than 
the  practical  use  of  this  information  and  communications  technology  in  routine 
health  care. 

The  lack  of  information  on  costs  and  benefits  of  telehealth  poses  problems  for 
those  intending  to  acquire  and  use  the  technology,  who  will  need  such  data  to 
assist  their  decisions.  Much  of  the  information  sought  will  need  to  come  from 
assessment  of  the  telehealth  application. 

In  1997  the  AHFMR  developed  and  published  a guideline  for  assessment  of 
telehealth  applications  2.  That  document  was  prepared  because  of  the  growing 
interest  and  commitment  to  use  of  telehealth  in  Alberta  at  that  time  and  the  need 
of  Regional  Health  Authorities  (RHAs)  and  others  for  advice  on  how  to  deal  with 
these  developments.  The  guideline  considered  assessment  of  telehealth 
applications  in  terms  of  the  attributes  Specification,  Performance  measures. 
Outcomes,  Summary  measures,  Operational  considerations  and  Other  issues.  A 
health  technology  assessment  (HTA)  perspective  was  taken,  with  some 
consideration  of  Safety,  Efficacy,  Effectiveness  and  Economic  impact  as  they 
might  apply  to  telehealth. 

The  AHFMR  guideline  considered  different  phases  in  the  life  of  a telehealth 
application,  drawing  attention  to  the  need  to  consider  planning,  feasibility  and 
pilot  studies,  initial  routine  clinical  use  and  later  developments  as  the  telehealth 
program  matured.  However,  there  was  particular  emphasis  on  the  early  stages 
of  this  process,  including  development  of  a business  case,  reflecting  the  needs  of 
RHAs  and  others  at  that  stage  of  the  diffusion  of  telehealth  technology  in  the 
province. 

At  about  the  same  time  that  the  AHFMR  guideline  was  being  prepared,  the 
Alberta  Mental  Health  Board  (AMHB)d),  in  collaboration  with  five  hospitals  in 
four  RHAs,  had  put  in  place  a telepsychiatry  pilot  project  as  an  intended  first 
step  in  the  development  of  Telemental  Health  Services  throughout  Alberta. 
Results  of  an  assessment  of  the  pilot  project,  which  drew  in  part  on  advice  from 
the  AHFMR  guideline,  were  presented  in  a report  and  in  subsequent  articles  3-5 . 

The  pilot  project  led  to  the  establishment  of  a routine  telepsychiatry  service  in 
central  Alberta  and  subsequently  to  a Telemental  Health  (TMH)  program 

1 1 1 In  this  paper  Alberta  Mental  Health  Board  has  been  used  throughout  to  denote  both  the  present  Board 
and  its  predecessors,  which  had  different  titles. 
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throughout  the  province.  These  initiatives  have  been  associated  throughout  with 
ongoing  assessment  undertaken  by  the  AMHB.  Experience  with  the  initial 
period  of  routine  operation  was  considered  in  a joint  AMHB/ AHFMR 
assessment 6-8  and  a further  AMHB  study  has  compared  telepsychiatry  services 
with  face  to  face  consultations  9. 

The  AMHB  experience  with  telepsychiatry  provides  an  unusual  opportunity  to 
review  the  long  - term  development  of  a telemental  health  program  from  the 
pilot  stage  through  to  a province  - wide  network  providing  routine  services.  An 
aspect  of  some  interest  is  the  extent  to  which  assessments  of  telepsychiatry  and 
other  services  by  the  AMHB  have  followed  the  provisions  of  the  AHFMR 
assessment  guideline.  Three  aspects  of  this  issue,  which  are  considered  in  this 
paper: 

• From  the  perspective  of  those  formulating  the  assessment  guideline,  there 
is  interest  in  whether  the  HTA  - based  provisions  have  been  followed  so 
as  to  provide  appropriate  information  on  which  to  base  decisions  on 
Telemental  Health  Services.  What  was  the  quality  of  the  studies  that  were 
undertaken  and  what  are  the  concerns  with  any  limitations  in  the  data 
obtained? 

• From  the  perspective  of  the  AMHB  as  the  organization  responsible  for 
development  and  assessment  of  the  telehealth  service,  there  is 
consideration  of  the  extent  to  which  the  guideline  was  seen  as  realistic. 
Were  the  points  included  in  the  guideline  seen  as  valid  and  relevant  to  the 
telehealth  service?  Was  collection  of  suggested  data  items  feasible  and 
potentially  useful?  To  what  extent  has  the  guideline  been  helpful  during 
the  evolution  of  the  Telemental  Health  Service? 

• There  is  interest  in  how  these  local  assessment  initiatives  have  informed 
the  policy  and  planning  process  at  AMHB  and  in  the  Regions.  What  have 
been  the  effects  of  assessment  of  telepsychiatry  services  on  decision 
making? 
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Elements  covered  in  the  assessment  guideline 


The  elements  for  telehealth  assessment  covered  in  the  AHFMR  guideline  are 
outlined  in  Table  1.  The  Specifications  section  is  supplemented  in  the  guideline 
by  a separate  table  that  outlines  the  requirements  for  a business  case.  Attributes 
typically  considered  in  many  health  technology  assessments  are  covered  under 
Outcomes  and  Summary  Measures.  Implicit  in  measurement  of  these  attributes 
is  comparison  with  a non-telehealth  alternative. 

The  guideline  considered  efficacy  of  a telehealth  application  to  be  the 
performance  of  the  technology,  after  an  initial  Teaming  curve7  period,  under 
carefully  applied  and  monitored  protocols,  such  as  those  used  in  pilot  projects. 
Effectiveness  relates  to  how  well  the  technology  will  perform  after  it  has  been 
adopted  for  routine  use.  It  was  noted  that  determination  of  effectiveness  would 
require  longer  term  follow  up  and  appraisal. 

The  other  elements  in  the  table  relate  to  important  local  issues  that  are  likely  to 
have  a major  effect  on  the  degree  of  success  of  a telehealth  application. 

The  guideline  included  a more  detailed  table  relating  specifically  to 
telepsychiatry  services.  That  table  has  provided  the  basis  of  the  summary  of 
AMHB  experience  given  here. 
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Table  1 : Elements  for  assessment  of  telehealth  applications 


Element 

Attribute 

Examples  of  items  for  the  telehealth 
application  and  non  - telehealth  alternative 

Specification 

Key  operating 
characteristics 

• Technology 
description 

• Assurance  of 
continuity 

• Health  service  description 

• Equipment,  carrier,  support 

• Personnel,  time 

• Performance  of  equipment,  training, 
maintenance 

Performance 

measures 

Time 

• Set  up  and  booking  time,  travel 

• Consultation,  intervention 

Quality 

• Image,  sound  and  service  quality 

Cost 

• Equipment,  training,  maintenance 

• Transmission,  travel,  staff 

Outcomes 

Safety 

E.g.  potential  adverse  effects  on  patient 
management  decisions  and  through  delayed  or 
missing  information 

Efficacy 

E.g.  Short  term  measures  of  timeliness, 
diagnostic  accuracy 

Effectiveness 

E.g.  Measures  of  length  of  hospital  stay, 
numbers  of  prescriptions,  repeat  consultations, 
satisfaction  with  service 

Summary 

measures 

Cost  effectiveness 

Cost  comparison 

Operational 

considerations 

Acceptability  to: 

• Patients 

• Health  care 
professionals 

• Managers 

Ease  of  use,  convenience,  cost  to  budget,  time 
savings 

Access 

Availability  to  population,  potential  increase  in  j 
clients,  potential  decrease  in  other  services 

Other  issues 

E.g.  Scheduling  of 
services, 
confidentiality, 
educational  benefits 

Source:  reference  2 
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Evaluation  requirements  for  the  Telemental  Health 
Service 


Growth  of  the  Telemental  Health  Service  has  taken  place  in  stages,  as  indicated 
in  Tables  2 and  3.  Somewhat  different  program  and  evaluation  requirements 
faced  the  AMHB  at  each  stage,  as  summarized  in  Table  4. 

Table  2:  Numbers  of  AMHB  telehealth  sites 


Fiscal  year 

Number  of  AMHB  telehealth 
sites 

Number  of  sites  added 
by  year 

1996/  97 

6 

6 

1997/98 

6 

0 

1998/99 

12 

6 

1999/00 

17 

5 

2000/01 

22 

5 

Table  3:  Numbers  of  sessions  in  the  telemental  health  service 


Completed 

clinical 

consults 

Educations 

sessions 

Administrative 

meetings 

Other 

1996/ 97 

ill 

1 

6 

9 

1997/98 

263 

7 

8 

10 

1998/99 

320 

4 

36 

42 

1999/00 

484 

45 

125 

78 

2000/01 

702 

88 

166 

37 
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Table  4:  Evolving  evaluation  requirements  from  the  telemental  health  service 


Stage 

Evaluation  requirements 

Pilot  project 
“is  this  feasible?" 
“should  we  exist?” 

Program  efficacy 
Process  details 

Initial  cost  analysis  - comparison  with  a traveling  psychiatrist 
Opinions  /satisfaction  for  patients  and  health  professionals 

Initial  routine  use 
“can  it  be  sustainable” 
“can  we  continue  to  exist” 

Program  effectiveness  , initial  indications  of  patient  outcome 
Level  of  use  and  access 
More  detailed  cost  analysis 
Influence  of  local  practice 

Expansion 

“how  should  it  develop 
further?” 

“should  we  exist  and 
expand?” 

Demonstration  of  need  by  the  participating  communities 
Appraisal  of  multi-application  use 
Access  for  under  serviced  communities 

Mature  network,  planning  for 
the  future 

“is  the  service  effective, 
efficient  and  equitable? 

“what  new  services  should 
exist?” 

Sustainability  - cost  of  equipment  and  network,  non-cost  issues 
such  as  referral  patterns 
Volume  of  service  by  receiving  sites 
New  technology  - effectiveness,  cost 
New  services  - benefit,  cost,  reach,  suitability,  satisfaction 
Ability  to  support  AMHB  programs  and  services  more  broadly 
Outcome  measures 

Evaluation  of  the  quality  of  the  service  applied  to  all  phases 
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AMHB  ASSESSMENT  ACTIVITIES  AND  THE  GUIDELINE 

Framework 


The  AMHB  assessment  experience  during  development  of  the  Telemental  Health 
Service,  including  the  various  stages  outlined  in  Table  4,  is  summarized  in 
Appendix  A.  The  tables  in  the  Appendix  are  based  on  based  on  the 
'Telepsychiatry  assessment  " table  from  the  AHFMR  guideline  2. 

Comments  on  the  AMHB  assessments  are  now  presented  from  the  perspectives 
of  the  guideline,  the  AMHB  program  and  decision-makers. 
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Guideline  Perspective  on  the  AMHB  Assessments  - 
What  has  been  Measured? 


From  the  perspective  of  the  guideline,  the  items  listed  under  Specification  and 
Other  issues  have  been  dealt  with  thoroughly  in  the  AMHB  assessments  3 6 and 
subsequent  program  management.  An  important  point  covered  in  the  guideline, 
though  not  listed  in  Table  1,  was  the  need  for  a person  to  coordinate  the  service. 
This  was  also  well  covered  in  the  assessments.  Acceptability  to  managers  has 
not  been  evaluated  formally,  though  there  has  been  frequent  contact  with 
managers  who  are  given  the  opportunity  to  provide  suggestions  about  the 
Telemental  Health  Service. 

Under  Operational  considerations,  acceptability  to  health  professionals  and 
patients  has  been  covered  throughout  using  survey  approaches  and  continues 
periodically.  Numbers  and  distribution  of  patients  have  been  obtained 
throughout  to  give  a useful  picture  of  the  extent  of  use  of  the  service.  A 
limitation  is  that  there  appear  to  be  no  comparative  data  on  patients  receiving 
face  to  face  consultations,  so  that  the  overall  place  of  the  TMH  service  in  the 
province  is  still  not  entirely  clear. 

Under  Performance  measures,  most  of  the  time  elements  have  been  measured  or 
considered  throughout.  The  data  relating  to  patients  and  families  were  obtained 
on  the  basis  of  fairly  basic  estimates  and  are  currently  collected  to  meet 
comparatively  limited  Alberta  We//net  requirements. 

Quality  issues  have  been  addressed  throughout,  including  satisfaction  with 
consultations,  with  at  least  some  comparison  with  face  to  face  consultations 
(opinions  from  surveys). 

Most  cost  data  elements  were  available  throughout  and  are  currently 
incorporated  in  routine  financial  records.  Only  limited  cost  information  is 
available  on  patient  and  family  items.  Information  is  made  available  to  Alberta 
We//net  regarding  patient  travel  costs  saved,  but  this  is  self  - reported 
information  and  is  collected  by  the  referring  site  rather  than  by  AMHB. 

With  Outcomes  there  are  some  limitations,  from  a guideline  and  Health 
Technology  Assessment  perspective. 

Safety  aspects  were  not  addressed  in  any  detail  throughout  the  assessments. 
However,  while  there  was  no  formal  measurement,  experience  and  data  from  the 
program  informs  several  of  these  points.  Responses  from  surveys  of  patients  and 
health  professionals  undertaken  during  the  pilot  and  routine  use  assessments 
suggested  that  the  service  was  well  accepted  and  that  misunderstood  advice  did 
not  appear  to  be  a significant  problem.  Consequences  of  missed  appointments  in 
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the  absence  of  telehealth  was  a matter  that  received  some  consideration,  albeit  in 
qualitative  terms.  Removal  of  travel  - associated  risks  for  both  patients  and 
psychiatrists  was  also  recognized. 

In  general,  there  would  be  high  confidence  that  safety  issues  were  not  of  major 
significance  for  this  telehealth  application. 

With  efficacy,  short  term  measures  of  satisfaction  with  the  service,  quality,  and 
timeliness  were  obtained  during  the  pilot  project  and  appeared  reasonable  for 
the  initial  stages  of  the  program.  Similar  sorts  of  information  were  obtained 
during  routine  use  and  usefully  confirmed  the  experience  of  the  pilot  project. 
However,  emerging  limitations  at  that  stage  were  the  absence  of  any  health 
status  measures,  other  than  survey  opinions  from  physicians,  and  also  the 
absence  of  comparative  data  from  face  to  face  consultations. 

Effectiveness  has  been  dealt  with  through  a small  benchmark  study  9 and  a 
randomized  controlled  trial  is  planned,  with  some  patients  now  enrolled. 

Routine  measurement  of  patients'  mental  health  status  has  now  been  started. 
Some  suggested  surrogate  measures,  such  as  drug  consumption  and  length  of 
hospital  stay,  have  not  been  feasible  though  some  consideration  was  given  to 
hospitalization  of  clients  during  assessment  of  the  routine  service.  In  general, 
there  still  seems  limited  information  on  the  comparative  performance  of 
telepsychiatry  and  face  to  face  consultation,  so  that  in  a strict  sense  level  of 
effectiveness  is  not  yet  established. 

With  Summary  measures,  the  first  two  assessments  included  break  even 
analyses,  comparing  the  costs  of  telehealth  with  those  of  face  to  face  consultation. 
Cost  effectiveness  has  not  yet  been  addressed  and  will  await  availability  of 
appropriate  comparative  data. 
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AMHB  Program  Perspective  on  the  Guideline:  What  is 
Relevant  and  Achievable? 


From  the  AMHB  perspective  the  relevance  and  usefulness  of  the  guideline  are 
influenced  by  the  realities  of  operating  a practical  telehealth  program  within  the 
Alberta  health  care  system.  Aspects  that  have  presented  particular  difficulty  are 
the  availability  of  data,  especially  outcomes  data,  and  the  increasing  complexity 
of  the  Telemental  Health  Service.  Another  practical  consideration  is  the  level  of 
resources  available  to  undertake  assessment  activities. 

Safety 

Measurements  of  the  variables  identified  in  the  AHFMR  framework  were  not 
performed  by  AMHB.  It  may  not  be  possible  to  gather  the  necessary  information 
to  report  on  these  suggested  measures.  For  example,  if  a client  misunderstands 
received  advice  it  may  not  be  possible  to  attribute  this  to  a telehealth-related 
factor.  An  aspect  that  has  been  considered  by  AMHB,  which  was  not  addressed 
in  the  assessment  guideline,  was  inclusion  of  a measure  of  adverse 
outcomes/ critical  incident  review. 

Effectiveness,  longer  term  measures 

Some  long-term  measures  identified  in  the  framework  do  not  have  a directly 
comparable  evaluation  activity  performed  by  the  Telemental  Health  Service. 
These  measures  include  participant's  time,  length  of  hospital  stay,  and  numbers 
of  prescriptions.  Some  of  these  measures  would  be  difficult  to  incorporate  as 
routine  evaluation  activities. 

There  is  general  information  on  the  length  of  a clinical  session  and  time  billed  by 
the  consulting  clinician  but  there  is  uncertainty  regarding  the  time  taken  by 
clients  or  other  participants  in  the  telehealth  process. 

Two  of  the  measures  suggested  in  the  framework,  length  of  hospital  stay  and 
number  of  prescriptions,  may  not  lend  themselves  to  the  Alberta  model  of  a 
Telemental  Health  Service  as  the  necessary  information  may  not  be  directly 
available  to  the  service  providing  telepsychiatry  consults.  Hospital  stay  data  are 
not  readily  available  to  the  Telemental  Health  Service. 

Specific  treatment  information  such  as  numbers  of  prescriptions  is  available  to 
the  client's  attending  physician  and  may  not  be  available  to  the  Telemental 
Health  Service.  Efforts  to  record  prescriptions  and  use  of  drugs  were  made 
during  the  earlier  telepsychiatry  assessments  but  consistent  data  could  not  be 
obtained.  However,  current  practice  is  to  record  for  each  consult  whether  any 
change  to  the  current  prescription  was  made  or  whether  there  was  a new 
prescription. 
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Cost  and  cost  effectiveness 

Cost  analysis  studies  can  be  difficult  to  perform  on  a service  that  is  expanding 
and  changing.  Cost  comparisons  of  telepsychiatry  with  the  alternative  were 
performed  as  part  of  the  pilot  project  and  the  assessment  of  routine 
telepsychiatry  services.  These  analyses  were  helpful  in  demonstrating  the 
feasibility  of  the  service. 

Any  extension  of  cost  studies  to  cost  effectiveness  measures  has  been  hampered 
by  the  absence  of  reasonable  quality  data  on  effectiveness  of  Telemental  Health 
Services.  A further  issue  that  has  emerged  as  the  AMHB  telehealth  program  has 
grown  is  the  increasing  complexity  of  the  network,  which  poses  challenges  for 
cost  effectiveness  studies.  At  this  stage,  a way  forward  would  be  to  perform  cost 
effectiveness  analysis  on  portions  of  the  service;  for  example,  the  Dual  Diagnosis 
Recovery  Service  that  was  introduced  in  August  2000.  A series  of  cost 
effectiveness  studies  would  seem  more  practical  than  attempting  to  consider  the 
Telemental  Health  Service  as  a whole. 
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What  is  the  Usefulness  of  the  Assessment  Framework? 


The  AHFMR  framework  focused  particularly  on  the  planning  and 
implementation  of  a new  telehealth  program.  From  the  perspective  of  an 
operational  telehealth  service,  the  framework  may  not  incorporate  all  necessary 
evaluation  criteria.  However,  the  framework  also  specifically  addresses  long 
term  issues,  especially  effectiveness  and  cost  effectiveness. 

The  framework  has  been  useful  to  the  Telemental  Health  Service  in  giving 
guidance  on  approaches  to  demonstrate  the  operation  and  usefulness  of  a 
telehealth  application.  This  has  been  valuable  in  terms  of  informing  stakeholders 
in  the  Telemental  Health  Service,  including  those  with  senior  administrative  and 
political  roles. 

Evaluation-  related  activities  undertaken  by  the  Telemental  Health  Service  differ 
in  some  areas  from  those  suggested  in  the  assessment  framework.  The  main 
reason  that  the  evaluation  activities  do  not  parallel  those  suggested  in  the 
framework  is  the  changing  evaluation  requirements  noted  in  Table  4.  Different 
evaluation  approaches,  not  captured  in  the  framework,  may  be  necessary  as  a 
service  grows.  As  the  service  has  expanded  the  information  that  is  reported 
internally  and  externally  has  changed.  Much  of  the  information  routinely 
collected  in  the  expanded  network  is  to  meet  reporting  requirements  and  assist 
quality  management,  rather  than  being  directed  to  specific  assessment  protocols. 

Changes  may  be  required  to  the  assessment  framework  to  extend  its  relevance  to 
an  operational,  large  scale,  telehealth  program.  However,  some  of  the  issues 
facing  the  AMHB  suggest  that  even  a modified  framework  might  have 
limitations  as  a worthwhile  aid  to  management. 

A reality  facing  the  Telemental  Health  Service  (and  most  other  telehealth 
programs)  is  that  resources  for  assessment  and  other  activities  do  not  grow  as  the 
volume  of  activity  increases.  Resources  available  to  undertake  some  of  the 
outcome  studies  visualized  in  the  framework  remain  very  limited.  There  is 
increasing  interest  in  the  TMH  and  the  number  of  requests  for  information. 

Many  data  requests  from  physicians  and  administrators  seek  detailed 
breakdowns  of  services  performed.  These  approaches  for  information  put 
further  demands  on  resources. 

The  increased  complexity  of  the  Telemental  Health  Service  presents  problems  for 
those  seeking  to  apply  an  instrument  such  as  the  AHFMR  framework,  which 
essentially  applied  to  a single  application  and  (by  inference)  to  a limited  number 
of  sites.  The  TMH  now  includes  a range  of  applications  - clinical,  educational, 
and  administrative  - and  a large  number  of  sites  throughout  the  province  and 
also  outside  Alberta.  Approaches  suggested  in  the  framework  are  helpful  in 
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considering  and  assessing  specific  areas  of  the  program,  for  example  in 
developing  a business  case  approach  for  new  applications  and  for  new  partners. 
Approaches  to  evaluation  of  the  service  as  a whole  would  need  to  be  developed 
and  would  be  complex  and  require  significant  assessment  resources.  A further 
area  of  complexity  is  that  the  AMHB  interacts  through  the  Telemental  Health 
Service  with  many  other  bodies,  including  Alberta  We/Inet  and  RHAs.  Many 
groups  are  involved  with  different  sites  and  applications  and  there  will  often  not 
be  common  standards. 

Some  current  and  potential  evaluation  and  data  collection  activities  of  the 
Telemental  Health  Service  are  summarized  in  Table  5. 

Table  5:  Potential  evaluation  and  data  collection  activities 


Data  to  support  ongoing  operation 

Growth  projections  together  with  budgetary  information  to  determine  if  changes  are  required 
to  the  service  to  ensure  sustainability.  For  example,  what  is  the  combined  network  cost  for  all 
sites  and  are  there  alternatives  that  would  reduce  these  costs. 

Demonstration  that  Telemental  Health  is  meeting  the  goals  defined  in  the  AMHB  business 
plan,  specifically  the  ability  of  TMH  to  improve  the  delivery  of  accessible,  effective,  quality 
mental  health  services  in  Alberta. 

Long  term  measures 

Cost  comparison/cost  effectiveness  evaluation  of  specific  subsections  of  the  overall  service. 
Determination  of  the  impact  of  telepsychiatric  services  on  the  health  care  system 

Retaining/recruiting  specialists 

How  to  recruit  and  retain  specialists,  and  sub  specialists. 

Other  evaluation  activities 

Urgent  consults  - outcome,  or  other,  measures  on  this  subsection  of  clients  served  by 
telepsychiatry. 

Information  for  other  telehealth  services 

How  to  pick  a new  site  or  new  service 

Which  variables  have  the  largest  impact  on  the  success  of  a new  site  e.g.,  physician  support, 
staff. 
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Decision  - makers’  Perspective  on  the  Assessments: 
What  Information  has  Helped,  What  is  Needed? 


Data  from  the  initial  assessment  were  used  to  obtain  the  commitment  of 
AMHB  to  progress  from  pilot  project  to  routine  (established)  services. 
Demonstrated  success,  from  the  assessment  of  initial  routine  services,  led 
to  approval  for  funding  of  additional  sites. 

The  pilot  project  introduced  the  concept  of  health  technology  assessment 
to  the  AMHB.  A province-wide  inventory  of  research  methodologies  in 
use  also  included  HTA.  The  assessments  provided  information  to  the 
AMHB  on  the  value  of  evaluation  and  the  need  for  timeliness  in  obtaining 
and  considering  assessment  results.  The  early  assessments  also  addressed 
issues  of  affordability  and  sustainability  before  they  became  more 
generally  canvassed  in  the  province.  The  relevance  of  the  AMHB 
approach  has  been  confirmed  and  maintained. 

While  not  directly  relevant  to  this  report,  the  AMHB  assessments  also 
provided  a foundation  for  further  evaluation  and  research  now  being 
undertaken  across  the  province. 

The  following  discussion  summarizes  other  ways  in  which  findings  from 
local  evaluation  have  been  applied  to  decision  making.  Some  of  these 
activities  relate  directly  to  points  made  in  the  assessment  guidelines, 
others  have  emerged  from  various  program  requirements. 

Evaluation  activities  as  required  by  Alberta  We//net  (ongoing) 

Alberta  We//net  reporting  provisions  were  introduced  in  1999.  Monthly  and 
quarterly  reports  are  submitted. 

The  monthly  activity  report  quantifies  the  number  of  transactions  (each  event  at 
each  participating  telehealth  site  is  a transaction)  and  the  number  of  attendees  at 
each  site.  Each  health  authority  is  responsible  for  submitting  a total  number  of 
transactions  and  participants  for  each  telehealth  site  on  a monthly  basis.  The 
quarterly  report  is  more  extensive,  with  three  main  areas  of  measure  - access, 
quality,  and  cost/  cost  benefit.  The  report  structures  are  shown  in  Table  6. 
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Table  6:  Alberta  We//net  telehealth  report  structures 


Monthly  reports 

Conference  Type 

Clinical 

Learning 

Administration 

Other 

Report  Total: 

Quarterly  reports 


Professional 

Count 

# 

# 

# 

# 

# 


Patient 

Count 

# 

# 

# 

# 

# 


Family 

Count 

# 

# 

# 

# 

# 


Access; 

a.  Average  wait  time  (computed  for  new  clients,  follow  up  clients,  and  % change  since 
previous  quarter) 

b.  Number  of  sessions  (by  clinical,  learning,  administrative) 

c.  Number  of  attendees  (also  % change  since  previous  quarter) 

d.  Number  of  new  programs,  with  a text  description  of  the  new  programs. 

Quality: 

a.  Number  of  satisfaction  surveys  completed 

b.  Number  of  participants  (by  clinical,  admin,  learning,  other,  for  clinical  attendees  are 
broken  out  by  professional,  patient,  family) 

c.  Per  cent  satisfied 

d.  Comments 

Cost/  Cost  Benefit 


Travel  avoided  (km) 

T ravel  cost  avoided  ($) 

Travel  time  avoided  (days,  hours,  min) 

Operational  costs  (Infrastructure,  Human  resources  costs,  Operational  costs) 

The  submission  of  these  reports  is  a requirement  and  many  of  these  measures  do 
not  inform  internal  decision-making.  AMHB  is  presently  working  with  other 
RHAs  and  Alberta  We//net  to  help  improve  the  reporting  process.  Some 
measures  submitted  to  Alberta  We//net  are  used  for  AMHB  evaluation  and 
administrative  needs,  including  preparation  of  reports  for  its  Board. 

For  sites  with  lower  numbers  of  consults,  use  of  alternative  services  is 
investigated  and  requirements  assessed  to  determine  if  the  existing 
videoconferencing  equipment  can  meet  the  demands. 

Wait  list  information  informs  the  need  to  recruit  more  consulting  physicians. 
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Routine  service  evaluation  results  were  incorporated  into  a business  case  used  in 
application  for  equipment  funding  from  Alberta  We//net.  Quarterly  evaluation 
statistics  are  used  in  combination  with  information  from  RHAs  to  demonstrate 
viability  and  growth  of  telehealth. 

Other  data  are  used  for  formal  and  informal  presentations  and  for  proposals. 

Evaluation  activities  as  required  by  Alberta  Mental  Health  Board 
(ongoing) 

Information  is  used  for  the  preparation  of  funding  proposals, 
presentations  to  internal  and  external  audiences,  and  drafting  of  reports  as 
required. 

Financial  analysis  (ongoing) 

This  includes: 

• Cost  of  network  - investigation  of  alternative  means  of  providing 
videoconferencing. 

• Cost  of  technology  to  investigate  lower  -cost  equipment  for  service 
delivery. 

• Sustainability  of  services  - use  of  a formal  business  case  to  justify 
allocation  of  resources  to  new  services. 

General  Practitioner  opinions 

A GP  satisfaction  survey  was  undertaken  in  2000,  considering  impact  of 
service  on  ability  of  physicians  to  treat  clients  in  the  community. 

Investigation  of  shared  care  model  of  service. 

Tele-education  assessment  finding  that  tele-education  sessions  are  not 
reaching  this  target  audience  (GPs). 

Comparison  of  telepsychiatry  with  conventional  consultation 

A benchmark  study  (2000)  was  used  to  provide  comparison  of 
telepsychiatry  with  face  to  face  alternative  and  give  assurance  that  this  is 
not  a second  best  service.  This  has  helped  to  establish  the  use  of  telehealth 
technology  in  the  community  mental  health  service. 

Client  opinions 

Client  telephone  interviews  in  1999  regarding  the  service  preferences  e.g., 
would  you  rather  use  telepsychiatry  than  wait  to  see  a psychiatrist  in 
person?"  Positive  response  indicated  the  suitability  of  the  service  to  this 
client  population.  Surveys  are  continuing  on  sporadic  basis,  as 
appropriate. 
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Client  follow  up  interviews  in  2000  were  undertaken  1 to  4 months  post 
consult.  Measures  included  self-reported  health  status  and  questions 
indicating  that  telepsychiatry  provided  the  mental  health  service  they 
required  and  an  overall  satisfaction  with  the  service. 

Telepsychology 

Comparison  of  psychological  test  scores  performed  face-to-face  and  by 
videoconference  to  determine  if  videoconferencing  is  a reliable  and  valid 
medium  for  the  assessment  of  cognitive  functioning.  Results  to  date  for  a 
geriatric  population  show  no  statistically  significant  difference  between 
face-to-face  and  videoconferencing  on  most  measures.  A trend  toward 
marginally  stronger  performance  in  session  two,  regardless  of  testing 
format  (i.e.,  practice  effect),  was  noticed. 

Results  have  been  used  to  demonstrate  the  viability  of  administering 
psychological  tests  using  videoconferencing  technology. 

Children’s  mental  health  service  (2001) 

Indicates  that  telepsychiatry  is,  for  the  most  part,  appropriate  for  this 
client  population. 
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Opportunities  for  Change  and  Limitations  to  Routine 
Assessment  of  Telehealth  Services 


From  the  experience  of  the  AMHB,  the  assessment  framework  has  some  useful 
features  and  was  a helpful  guide,  particularly  in  the  earlier  stages  of  the 
Telemental  Health  Service.  It  has  been  less  useful  as  the  service  has  developed 
further.  Some  specific  suggestions  for  change  to  the  guideline  are  to  include 
subsections  on  specific  types  of  service  and  to  identify  the  most  important  or 
relevant  indications  and  outcome  measures.  A possible  problem  here  is  that  of 
matching  a generic  assessment  guide  to  the  specific  needs  of  a telehealth 
network,  which  might  require  a 'customized7  approach,  while  retaining  essential 
assessment  principles. 

An  addition  to  the  framework  that  is  suggested  by  the  AMHB  is  some  measure 
of  adverse  outcomes/ critical  incident  review  as  a component  of  assessing  safety 
of  the  telehealth  application.  A challenge  in  applying  such  provision  might  be 
the  difficulty  in  obtaining  details  and  clarifying  responsibilities  in  the  off  site 
situation. 

The  AMHB  would  wish  the  framework  to  reflect  a realistic  expectation  of  the 
evaluation  activities  that  can  be  undertaken  by  an  operational  telehealth 
program.  This  in  turn  points  to  some  challenges  facing  assessment  of  established 
telehealth  networks  that  are  likely  to  apply  in  many  health  care  systems.  Long 
term  data  collection  and  availability,  resources,  network  complexity,  and 
decision  makers7  needs  are  all  issues  that  have  emerged  during  the  development 
of  the  Telemental  Health  Service. 

Routine  data  collection  may  be  well  established,  as  is  the  case  with  the  AMHB 
program,  but  is  unlikely  to  include  items  that  would  be  needed  for  specific 
assessment  purposes.  Health  related  quality  of  life  measures  would  be  an 
example.  In  such  areas,  data  collection  is  more  likely  to  be  linked  to  conduct  of  a 
specific  study,  when  resources  can  be  found  to  put  it  in  place.  Other  types  of 
data,  such  as  use  of  drugs,  may  lend  themselves  more  obviously  to  routine 
collection,  but  for  various  local  reasons  may  not  be  able  to  be  captured  reliably, 
as  has  been  the  case  with  the  Telemental  Health  Service.  Further  possible 
limitations  are  the  scope  of  administrative  data  and  restricted  access  to  them  that 
may  apply  more  generally  within  a health  care  system. 

Resources  for  assessment  at  the  local  level  are  a real  difficulty.  Even  with  the 
AMHB,  which  has  maintained  a continuous  assessment  program  throughout  the 
life  of  the  Telemental  Health  Service,  there  are  limits  to  evaluations  that  can 
realistically  be  undertaken. 
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The  complexity  of  a routine  telehealth  network  such  as  the  Telemental  Health 
Service  poses  challenges  for  assessment,  and  further  work  on  economic  and  other 
evaluation  approaches  seems  desirable.  Certainly,  there  seem  to  be  definite 
limits  to  the  usefulness  of  a generic  assessment  framework  in  this  situation. 

Parts  of  a complex  telehealth  network  can  still  be  usefully  assessed  using  the 
framework  as  a guide,  but  overall  appraisal  of  such  a system  seems  to  require  a 
somewhat  different  approach. 

Finally,  resources  and  topics  for  assessment  will  in  practice  need  to  be 
considered  in  the  light  of  overall  needs  of  decision-makers  in  the  health  care 
system.  Some  of  these  needs,  such  as  data  on  access  and  level  of  use,  may  be 
closely  tied  to  administrative  or  political  decisions  and  may  take  priority  over 
data  on  outcomes  of  telehealth  services. 
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Appendices 


Appendix  A:  Comparison  of  TMH  Evaluation  Activities 
with  AHFMR  Telepsychiatry  Evaluation 
Framework 

The  basis  for  comparison  in  the  following  tables  is  the  detailed  list  of  suggestions 
for  a telepsychiatry  application  given  in  the  AHFMR  guideline  2.  AMHB 
evaluation  experience  in  different  areas  is  summarized  for  the  different  phases  of 
the  Telemental  Health  Service  program  - pilot  project,  initial  routine  use,  1998 
expansion  and  present  service. 

Abbreviations 

AHP:  Alberta  Hospital  Ponoka 
TMH:  Telemental  Health  Service 
DSM IV:  Diagnostic  and  Statistical  Manual 
GAF:  Global  Assessment  of  Functioning 
RHA:  Regional  Health  Authority 
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Table  7:  Comparison  of  specification  details 
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Table  7:  Comparison  of  specification  details  (cont’d) 
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Table  8:  Performance  measures  - time 
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Table  10:  Performance  measurement 
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Table  12:  Outcomes  - efficacy 
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Table  13:  Outcomes  - effectiveness 
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Table  15:  Operational  considerations 
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